Papers with contrastive learning method
KE-GCL: Knowledge Enhanced Graph Contrastive Learning for Commonsense Question Answering (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for commonsense question answering lack effective representations of knowledge graphs. |
| Approach: | They propose a Knowledge Enhanced Graph Contrastive Learning model by incorporating contextual descriptions into QA pairs and adopting a graph contrastive learning scheme. |
| Outcome: | The proposed model outperforms existing methods consistently on two benchmark datasets. |
RAG4ITOps: A Supervised Fine-Tunable and Comprehensive RAG Framework for IT Operations and Maintenance (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have improved the open-domain QA’s performance, but how to efficiently handle enterprise-exclusive corpora and build domain-specific QA systems are still not studied for industrial applications. |
| Approach: | They propose a general and comprehensive framework based on Retrieval Augmented Generation (RAG) and facilitate the whole business process of establishing QA systems for IT operations and maintenance. |
| Outcome: | The proposed framework achieves superior results on two kinds of QA tasks. |
Fine-grained Contrastive Learning for Definition Generation (2022.aacl-main)
Copied to clipboard
| Challenge: | Recent pre-trained transformer-based definition generation models lack effective representation learning to contain full semantic components of the given word, leading to under-specific definitions. |
| Approach: | They propose a novel contrastive learning method that encourages the model to capture more detailed semantic representations from the definition sequence encoding. |
| Outcome: | The proposed method could generate more specific definitions compared with state-of-the-art models. |
Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing topic models do not make full use of word co-occurrence information to model latent topics. |
| Approach: | They propose a novel short text topic modeling framework, Topic-Semantic Contrastive Topic Model, which uses augmented data and the data characteristic to learn the relations among samples. |
| Outcome: | The proposed framework outperforms state-of-the-art baselines regardless of the data augmentation availability, producing high-quality topics and topic distributions. |
Self-Guided Contrastive Learning for BERT Sentence Representations (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to derive sentence embeddings from pre-trained Transformers are unclear . a self-guided training method is used to fine-tune BERT in a supervised fashion . |
| Approach: | They propose a contrastive learning method that utilizes self-guidance to improve BERT sentence representations. |
| Outcome: | The proposed method is more effective than baselines on diverse sentence-related tasks and robust to domain shifts. |
Contrastive Learning of Sentence Embeddings from Scratch (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learn sentence embeddings with unlabeled data are limited due to copyright restrictions, data distribution issues, and messy formats. |
| Approach: | They propose a contrastive learning framework that trains sentence embeddings with synthetic data. |
| Outcome: | The proposed framework produces positive and negative annotations given unlabeled sentences and generates sentences along with their corresponding annotations from scratch. |
Contrastive Learning as a Polarizer: Mitigating Gender Bias by Fair and Biased sentences (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have highlighted social biases inherent in training data can lead models to learn and propagate them. |
| Approach: | They propose a contrastive learning method that uses anchor points to push further negatives and pull closer positives within the representation space. |
| Outcome: | The proposed method achieves state-of-the-art in the ICAT score on the StereoSet, a benchmark for measuring bias in models. |
CoSQA: 20,000+ Web Queries for Code Search and Question Answering (2021.acl-long)
Copied to clipboard
| Challenge: | Using deep neural networks to find codes is difficult . we present a dataset that includes 20,604 labels for natural language queries and codes . |
| Approach: | They introduce a contrastive learning method to enhance text-code matching . they find that CoSQA improves the accuracy of code question answering by 5.1% . |
| Outcome: | The proposed method improves the accuracy of code question answering by 5.1% and improves by 10.5% on a CodeBERT model. |
xMoCo: Cross Momentum Contrastive Learning for Open-Domain Question Answering (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to find relevant passages using sparse keywords are not effective for open domain question answering. |
| Approach: | They propose a new contrastive learning method for learning a dual-encoder model for question-passage matching using a large pool of negative samples. |
| Outcome: | The proposed method maintains large pool of negative samples and optimizes question-to-passage and passage-to question matching tasks. |
BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation (2023.findings-acl)
Copied to clipboard
Liyan Kang, Luyang Huang, Ningxin Peng, Peihao Zhu, Zewei Sun, Shanbo Cheng, Mingxuan Wang, Degen Huang, Jinsong Su
| Challenge: | Existing datasets focus on captions describing images or videos, which are not large and diverse enough. |
| Approach: | They propose a large-scale video subtitle translation dataset to facilitate multi-modality machine translation. |
| Outcome: | The proposed dataset is 10 times larger than the widely used *How2* and *VaTeX* datasets. |
Text Classification by Contrastive Learning and Cross-lingual Data Augmentation for Alzheimer’s Disease Detection (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for AD detection are too expensive and time-consuming to cover all potential patients. |
| Approach: | They propose a contrastive learning method to obtain effective text representations based on monolingual embeddings of BERT and a cross-lingual data augmentation method by building autoencoders to learn the text representation shared by both languages. |
| Outcome: | The proposed method outperforms other methods on a Mandarin AD corpus and achieves 81.6% detection accuracy. |
Isotropy-Enhanced Conditional Masked Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing non-autoregressive models with auto-regressing decoding paradigms have been used for various text generation tasks to accelerate inference but at the cost of generation quality to some extent. |
| Approach: | They propose to use Look Neighbors strategy to enhance learning of target token representations during training to achieve a good balance between inference speedup and generation quality. |
| Outcome: | The proposed models outperform current models on 4 WMT datasets and outperformed the current SoTA results. |
CLAD-ST: Contrastive Learning with Adversarial Data for Robust Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Cascaded approach is the most popular choice for speech translation, but lacks robustness when dealing with noisy inputs. |
| Approach: | They propose a cascaded approach that uses an automatic speech recognition model and a machine translation model to translate speech in one language to text in another language. |
| Outcome: | The proposed approach achieves significant gains of up to 3 BLEU scores in English-German and English-French speech translation without hurting the translation quality on clean text. |
Generalizable Implicit Hate Speech Detection Using Contrastive Learning (2022.coling-1)
Copied to clipboard
| Challenge: | Hate speech detection is challenging when there are insufficient lexical cues. |
| Approach: | They propose a contrastive learning method that pulls an implication and its corresponding posts close in representation space. |
| Outcome: | The proposed method improves on BERT and HateBERT benchmarks on three implicit hate speech benchmarks. |
PromptBERT: Improving BERT Sentence Embeddings with Prompts (2022.emnlp-main)
Copied to clipboard
Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, Qi Zhang
| Challenge: | Existing research shows that BERT and RoBERTa are poor in sentence embeddings due to static token embeddable bias and ineffective BERT layers. |
| Approach: | They propose a novel contrastive learning method for better sentence embeddings by using a template denoising technique. |
| Outcome: | The proposed method achieves 2.29 and 2.58 points of improvement compared to SimCSE and RoBERTa in the unsupervised setting. |
Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment (2025.emnlp-main)
Copied to clipboard
Jingcheng Deng, Zhongtao Jiang, Liang Pang, Zihao Wei, Liwei Chen, Kun Xu, Yang Song, Huawei Shen, Xueqi Cheng
| Challenge: | Experimental results demonstrate that our method significantly outperforms traditional contrastive learning approaches when using the same amount of data. |
| Approach: | They propose a new contrastive learning method built on embedding conditional probability distributions that integrates two tasks: information compression and conditional distribution alignment. |
| Outcome: | The proposed method outperforms traditional contrastive learning approaches and achieves comparable performance to state-of-the-art models when using the same amount of data. |